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Making Decisions Using Assessment Data in the Time of COVID-19 and Beyond 


his shift in mindset includes recognizing that summative 
assessment data are most useful when placed in appropriate 
context. Understanding the impact of the COVID-19 crisis on stu- 
dent learning is fundamental for being able to develop respon- 
sive supports that address the needs of all students, particularly those 
most affected by the disruptions to learning in the 2019-20 and 2020-21 
school years. Policymakers and education leaders can use valid, reliable 
data on student learning to identify inequities, support inquiry about 
systemic factors that contribute to inequity, and inform strategic alloca- 
tion of resources, services, and staff to accelerate learning for those 
who need it most. Meaningful assessment data are essential for policy- 
makers and stakeholders designating funding and determining eligibility 
requirements for programs that can impact student learning, such as 
state-supported professional learning resources and relief funding 
from sources such as the American Rescue Plan Act (ARP). And parents, 
educators, and the public need information about student learning in 
order to support their students’ academic progress. 


In response to flexibility offered by the federal Department of Education, 
many states developed plans for assessment in the 2020-21 school 

year that decouple summative assessment data from federal account- 
ability requirements and that offer alternatives in situations in which 
student participation in the state's traditional summative assessment 

is not viable. For these plans to be effective, educators will need to pay 
attention to data, including assessment results, beyond just the state's 
summative test. Furthermore, state and local education agencies will 
need to decide which data can provide meaningful information about 
student, school, and system performance. Accordingly, it is particularly 
important this year for states and districts to understand best practices 
for using and analyzing data to support student learning. 


In this brief, we answer some timely questions about how to think about 
assessment information at this unique moment, and we offer several 
suggestions and considerations about how to use assessment data to 
inform decisions about teaching and learning. 


Making Sense of Summative Assessment 
Data from Spring 2021 


Given all the disruption to student learning, can assess- 
ments from 2021 be considered valid? 


Validity is actually not a property of an assessment, but a property 

of the meaning given to, or the inferences made from, assessment 
scores.'* Because educators draw inferences from assessments to 
inform important student-, school-, and system-level decisions, they 
need to think through whether the conclusions they make and the 
actions they take in response to assessment results are justified by the 
properties of the test and consistent with what the test was designed to 
do. By themselves, teaching and learning disruptions do not invalidate 
assessment results. If well-designed assessments are administered 
under standardized conditions, then the results can increase our 
understanding of how effectively schools are preparing students to 
meet grade-level standards, even during the pandemic. For instance, 
assessment data can help show how well distance and hybrid learning 
strategies supported student learning during the pandemic. These 

data can reveal areas in which students need more support and inform 
discussions about practices that were introduced during the pandemic 
and may be worth continuing. 


Important education decisions should, whenever possible, be supported 
by multiple sources of information to ensure that they are sound and 
based on the fullest understanding possible of student learning experi- 
ences and outcomes. While 2021 assessment data can still be a helpful 
barometer of how well educators and schools supported students’ 
grade-level learning, it is not appropriate to use these data alone to 
make inferences about student success or school quality, particularly if 
such inferences are attached to significant decisions or consequences. 
To avoid drawing incorrect conclusions from assessment data about 
student success or school quality, policymakers and education leaders 
should consider lowering or removing any high stakes attached to 2021 
assessment results. They should also consider additional measures of 
student learning that will supplement summative assessment data to 
provide the richest picture of students’ academic progress. 
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Can assessments from 2021 be considered reliable? 


Reliability refers to the consistency of measurement from an assess- 
ment. In other words, would the same results be obtained if the assess- 
ment were given and scored again? As with assessment validity, if a 
well-designed assessment is given under standardized conditions, it 
can be expected to be reliable, even during a year of pandemic-related 
disruptions to learning. 


What if the assessment conditions aren't standardized, or 
if test security cannot be assured? 


In many communities, the pandemic has posed challenges to admin- 
istering a standardized, secure assessment. Even in communities in 
which schools reopened for in-person schooling in some capacity for 
the 2021-22 school year, families were given the option to remain in 
remote learning programs through the rest of the year. Consequently, 
some students will only have participated in testing remotely, which can 
pose challenges in terms of students’ equitable access to technology, 
bandwidth, and appropriate levels of support. 


If testing administration conditions reflect this sort of variation in 
access and, thus, are not standardized, educators and policymakers 
cannot be sure that assessments are measuring the same things for all 
students. If these challenges cannot be overcome, then it is likely that 
the assessment results will not be reliable. 


If different students had widely different access to 
learning the standards, shouldn't we throw out the 
assessment results? 


Opportunity to Learn (OTL) refers to the system inputs experienced by 
students in their learning, such as access to fully qualified teachers 

or quality learning materials. As in any year, when students’ learn- 

ing experiences have varied widely, assessment data may actually be 
measuring OTL much more than student knowledge, skills, and abilities. 
Because factors such as access to home broadband and computer 
hardware were gatekeepers to participation in distance learning for 
many students, it is essential to consider OTL data when making sense 
of and responding to inequities that emerge from the 2021 outcome data 
on student learning. 


So, while assessment results should not be disregarded, educators and 
others should collect and use OTL data in conjunction with summative 
assessment data and other data sources to inform inferences and 
action steps. Such data can be collected via surveys of parents or 
teachers, examination of teacher assignments, and reviews of resource 
allocation, among other approaches. Analyzing OTL data alongside 
assessment data will help ensure that the focus is on the systemic fac- 
tors that result in unequal assessment outcomes. 


Given that many educators have concerns about the 
validity and reliability of assessment data from 2021, why 
should we bother with summative assessment this year? 


When used in conjunction with accurate OTL data, high-quality sum- 
mative assessments are still an important source of information about 
student learning. This full set of data can be disaggregated and exam- 
ined as part of a broader root-cause analysis to understand inequities in 
inputs that produce inequitable student outcomes and inform improve- 
ment efforts that support higher achievement for all students. 


What Can Be Done If We Have Concerns 
About the Accuracy and Usefulness of 
2021 Assessment Data? 


Lower the Stakes 


In spring 2021, the United States Department of Education (ED) issued 
guidance that affirmed the importance of summative assessment but 
recognized the need to lower the stakes associated with summative 
assessment results.' Specifically, ED permitted states to exclude certain 
assessment results from federal accountability for the 2020-21 school 
year. Many states have taken advantage of this flexibility. Decoupling 
summative assessment data from federal accountability require- 

ments signals that the primary purpose of state assessment data is to 
measure learning, so state and local systems can use those assessment 
data to inform decisions about teaching and learning. 


Given that there are concerns about variation in assessment conditions 
and cautions about not using 2021 assessment data to draw inferences 
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about school quality and student performance, states and districts 
should follow ED’s lead and refrain from making high-stakes decisions 
based solely on assessment data. 


Use Multiple Measures to Understand Student Learning 


As policymakers wrestle with thorny decisions such as how ARP relief 
funds should be spent, they need a variety of information sources to 
shed light on resource needs and other investments that can support 
student learning. Summative data should never be the sole measure of 
student learning used to make decisions, particularly not in a year with 
major disruptions to teaching and learning. Using multiple measures can 
provide a more nuanced picture of the current status of learning. These 
measures can include OTL data; local assessment data; and qualitative 
data from students, teachers, and families about student experiences 
and learning. 


Collect and Use OTL Data 


In any year, OTL data provide important contextual information about 
the conditions in which students learn and the factors that contribute 
to student learning outcomes. For the 2020-21 school year, OTL data can 
include information about student access to the internet and computer 
hardware, access to appropriate books and learning materials, and 
hours of available instruction with qualified teachers. These data should 
be collected and viewed in tandem with assessment data to inform any 
inferences made about student learning. Analysis of OTL data alongside 
outcome data can also help identify inequitable inputs that may be 
reflected in summative results and inform decisions about providing 
appropriate resources to support accelerated learning. 


In most contexts, OTL is not static. Over the course of the 2020-21 

school year, for instance, many student learning experiences changed — 
often dramatically. Therefore, OTL data should be collected at key inter- 
vals to ensure the data reflect the lived learning experience of students. 


Leverage Interim Assessment Results 


In using multiple measures of student learning, interim assessment data 
can provide valuable additional information. In most years, interim data 
can be used in concert with summative assessment results. However, 


in 2021, some districts will opt to use local interim or benchmark 
assessments in place of summative assessment if they deem that 
administering state assessments is not viable. In either context, some 
of the considerations or limits on the inferences that can be made from 
summative assessment data this year also apply to data from interim 
assessments, given similar concerns about variation in assessment 
administration conditions. 


Additionally, when making inferences based on interim assessment 
data, it is important to consider the attributes of the specific interim 
measure selected: How well does it align to the standards? What content 
is underrepresented or omitted? For example, an interim assessment 
that focuses on reading comprehension cannot provide meaningful 
information about the status of broader 
literacy skills such as writing, speaking, 
and listening. It is also important to con- Defining 


sider the appropriateness of the interim Assessment 


assessment for students with disabilities 


Literacy and Data 
MiGlela) 


and the availability of necessary accom- 
modations and supports available to all 
students who take the assessment. 


“Assessment literacy is 
defined as the knowledge 
about how to assess what 
students know and can 
do, interpret the results 
of these assessments, 
and apply these results 
to improve student 
learning and program 
effectiveness.” 


Collect and Use Qualitative Data 
from Students, Families, and 
Teachers 


Authentic engagement of students, fami- 
lies, and teachers is essential for estab- 
lishing a more complete picture of student 
learning from which to make decisions, 
which is particularly important during the 
COVID-19 pandemic. These sources can 
provide essential qualitative data about 
how students experienced learning in Data-literate educators 
continuously, effectively, 
and ethically access, 


remote contexts, which can, in turn, be 
used to contextualize other measures and 


interpret, act on, and com- 
municate multiple types of 
data from state, local, class- 
room, and other sources 

to improve outcomes 

for students in a manner 
appropriate to educators’ 
professional roles and 
responsibilities.’® 


inform appropriate support. 


Valuing the data that students, families, 
and teachers provide about learning 
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not only improves understanding of students’ academic successes, 
challenges, and needs, but it can also support rebuilding school-home 
relationships and community engagement as students return to school. 


Pay Attention to Who Is Not in the Data Set 


Although flexibility from ED will remove the traditional penalty to schools 
or districts that do not achieve 95 percent participation in this year’s 
assessments, the requirement to administer the standardized assess- 
ments was still in place for spring 2021. As a result, schools should have 
made efforts to administer the required assessments. 


Despite those efforts, it is reasonable to expect that many, if not all, 
schools will have less than full participation. Local educators may be 
able to glean insights by identifying which students are missing from 
the spring 2021 assessment data set. For instance, identifying who is 
missing from the assessment data set may uncover inequities or point 
to actions needed to serve all students: Do untested students tend to 
share common features, such as race/ethnicity? Does the composition 
of untested students differ from the school as a whole, in terms of grade 
level, race/ethnicity, and past academic performance? A“yes” answer to 
either of these questions may be a flag that some groups of students 
face higher barriers to participating and engaging in school activities 
such as testing, and/or that more outreach or different communication 
is needed. 


seize the Opportunity to Build Stronger 
Data Practices 


Instead of reacting to concerns about the quality of assessment data 
during the COVID-19 pandemic with quick fixes, state and local systems 
can use this opportunity to establish sustainable data practices that can 
support improvement in teaching and learning and promote equitable 
outcomes for all students well beyond the pandemic. To do this, state 
and local leaders will need to articulate a clear vision for the local sys- 
tem of assessment and how it aligns to and supports goals for learning 
shared by the community, local educators, and state policymakers. 


Leaders must also engage teachers, principals, and local administrators 
in building and sustaining balanced and systemic assessment and data 
practices. The following investments can support the development of a 
strong culture of assessment literacy and data literacy (see sidebar) in 
which assessment data are used to highlight and address inequities and 
improve teaching and learning: 


» Build acommon base of assessment literacy, including an under- 
standing of the distinctions between different types of assessment, 
the purposes of these assessments, and the appropriate use of the 
data they yield. 


» Foster an expectation that analyzing data requires interrogat- 
ing biases and committing to a shift from a deficit mindset to an 
improvement mindset. 


» Commit to developing culturally responsive data literacy, which 
places an “explicit focus on both centering culture and equity, 
emphasizing specific data skills and dispositions that are needed to 
implement an inclusive and asset-based approach to data interpreta- 
tion and use.”® 


» Develop data use norms and practices that ensure all data-based 
decision-making includes meaningful consideration of multiple 
measures, including OTL data and qualitative data gathered through 
authentic stakeholder engagement. 


» Support teachers and administrators to apply learning about 
assessment practice and data use to their own context by allocating 
sufficient time and resources for collaboration and inquiry around 
assessment literacy and data literacy. 


Conclusion 


Although the administration of summative assessments in spring 2021 
looks different than most years, educators should resist the temptation 
to conclude that the 2021 standardized assessments will not provide 
any useful information. Rather, asking how the data and evidence col- 
lected can be used appropriately may point schools and districts to new 
solutions to address inequitable system inputs, uneven opportunities to 
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learn, inefficient allocation of program dollars or eligibility, and ineffec- 


tive outreach to students and families. 


The pandemic provides an opportunity to revisit the stakes and 
consequences associated with different assessments, as well as a 
chance to consider multiple measures to ensure a fuller picture of the 
performance of students, classrooms, schools, districts, and states. 


Such a broader understanding will be essential to informing the efficient 


investment of ARP funds, and resource allocation more broadly. Our 
hope is that educators will be able to use this period as an opportunity 
to re-evaluate their assessment practices and make important shifts 
toward strengthening assessment data-use practices, making indelible 
improvements to decision-making, equity, teaching, and — ultimately — 
student learning and success. 
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